DOCUMENT RESUME 



ED 046 348 



HF 001 037 



A UTHO p 

"H i *t* ij t 

INSTITUTION 

OUS i. ATE 
NO m E 



*DBS PRICE 
DESCRIPTORS 

IDENTIFIERS 



Eox, A, M. ; Brookshire, william K. 

Eefininq Effective College Teaching Using the Delphi 
Technique and Multiple Linear Regression. 

American Educational Research Association, 
Washinoton, P.C. 

Jan 7i' 

12p.; D aper presented ah the 1°7i Convention of the 
American Educational Research Association, New Yor v 
City 

eqRE Price MT-TO.fE HC-S3.2 0 

♦Coll^ae ^eachers, * ref ini t i ons , Effective 

Tea chi no, valuation, *Hiah.er education, Teacher 

Attitudes, Tpgcher Characteristics, Teaching fiuali*v 

♦Delphi Technique, Northern Colorado University 



APE? D AC T 



A modified form o f th« Delphi technique /as used 
within the s ?hool of education at the University of Northern Celorado 
to formulate a definition of effective college teachina. ?h° 
resulting definition contained Z major categories: (1) personal 

characteristics: frienlly, mature, enthusiastic; (?) professional 
qualities: attitudes, krowl^dae, preparation; (?) interpersonal 
relat ionshius: approachable, fair, honest; (4) technical skills of 
teaching: methods, grading, organization; and ( c ) communication 
skills: rapport, relevancy, open. These results were use* to develop 
2 5 fictitious profiles of faculty members. Participants were askel +0 
separate these 2F profiles into f sets accordina to those nost 
deserving of promotion. Multiple linear regression was used to 
analyze these judgments to determine the priori tv Placed on each o f 
+h° major categories. (Author/ AT) 



DEFINING EFFECTIVE COLLEGE TEACHING USING THE 
DELPHI TECHNIQUE AND MULTIPLE LINEAR REGRESSION 



CO 

• 4 " 

KN 

v£> 

4 “ 

O 

Q 



Fox, A.M. 

Bureau of Research 
University of Northern Colorado 



Brookshire, William K. 
North Texas State University 



Presenter to the 

American Educational Research Association 

1971 Convention 
New York City 



A, 

n 

X 



I o 
"ERIC 



U ft DtrAKrM(HtOf HlAltM (Dl/CAIIO^I 
ft WCltAM 

OfMCI Of IDUCATiO* 
r« s ooo/miat was kc« <fAcc*xfo 
UACTL1 AS aeckvi d r aou r m i ri*son oa 

OA&AHtf ATON OA'C'AAT(SG IT AO^TS o* 
V't*OA 0***C**5 S»ATtO 00 SOT %?Cf$ 
SAA ii ACfAESIHT OTTCiAl OM<IOHt*U 
CAT** AOS'TOA OA WOv«CT 



X 



Defining Effective College Teaching Using 
The Delphi Technique And Multiple Linear Regression 



Fox, A. M. 

Brookshire, William K. 

Short Abstract 

A modified form of the Delphi Technique was used within the School 
of Education at the University of Noi them Colorado to formulate a 
definition of effective college teaching. The resulting definition 
contained five major categories. Thes° results were us^d to develop 
twenty-five fictitious profiles of facul ty member s . Participants were 
asked to separate these twenty-five profiles into five sets according 
to those most deserving of a promotion. Multiple linear regression was 
used to pr.dlyze these judgments to determine the priority placed on 
each of the major categories of the definition of effective teaching. 




Tin re has been recently an increased concern about the effectiveness 
of teaching at the higher levels. In particular there is deep concern 
about the quality of teaching currently in practice on college campuses. 
College students are asking for an opportunity to evaluate their profes- 
sors. Many faculty members are becoming unhappy with a lock step salary 
scale, automatic promotions, and tenure. Instead, the faculty members 
desire a chance to prove themselves through some system of evaluation with 
the hope that an outstanding job on their part will be rewarded. 

There is also some pressure from outside the academic cor, m unity for 
evaluation of the effectiveness of college teaching. More and more the 
taxpayer is asking for evidence of value received for his tax dollar 
spent . 

That there is a demand for the evaluation of the effectiveness of 
college teaching is no longer a question. 

The question now before us is a professional question. One of 
how to accomplish this evaluation. If this professional question of the 
effectiveness of college teaching is to remain within the profession, we 
as educational researchers must work hard to find answers and outline 
strategems whereby the profession can set its own guidelines and standards 
for evaluation and in fact do the * valuation. 

One of the first difficulties one encounters when attempting research 
in the area of effective teaching is that of locating a widely accepted 
definition of effective teaching. 

As a part of i he efforts being made at the University of Northern 
Colorado to develop useful evaluative techniques and instruments a study 
was undertaken to formulate a definition of effective teaching and further 
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to determine the relative importance of the various elements of such a 
definition . 

Since the 1970 AERA convention brought out the fact that no one does 
a Delphi Study, we too used a modified version of a Delphi Study within 
the School of Fducation faculty at the University of Northern Colorado. 
Those interested in more information concerning the Delphi Techniqje 
are referred to the bibliography. There were 91 faculty members from 
the School of Education kind enough to participate in the study. 

The first round of the modified Delphi Study asked each faculty 
member to list what he considered to be the ingredients of effective 
college teaching. They were encouraged to feel free in their response 
and to be as complete in their definition as they desired. 

The returns from this first round were screened by the researchers 
in an effort to eliminate duplicate statements and to prepare a combined 
listing of all the statements in preparation for the second round. 

There were 195 different statements, phrases, or key w T ords, offered 
by the respondents in the first round. Examples of those statements 
are presented in Table I. 

These statements, phrases, and key words were numbered and used as 
elements of a definition of effective college teaching in the second 
round of the study. 

Each participant was asked in the second round to read through the 
195 elements and check those which he felt should be retained as part 
of a definition of effective college teaching. It was anticipated that 
some elements would appear very similar in meaning. In such cases the 
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respondent was asked to check the element he felt best and leave the 
other similar elements unchecked. Space was provided for persons to add 
any elements they had thought of since the first round. 

TABLE I 

EXAMPLE OF FIRST ROUND RESPONSES 

1. Ability to maintain discipline by being friendly and unders tand ing 
yet demonstrates authority by being a genuine, sincere person. 

.1, Is warm, friendly and flexible. 

3. Inspiring. 

A. Enthusiasm. 

b. Integration of concepts. 

(i. Think of students as individuals. 

7. Empathize. 

tf. Respect for students' viewpoints. 

') . Allows students to disagree. 

L 0 . Knowledge of the subject. 

11. He starts where the students are. 

12. Involve the students in creative learning. 

13. An ability to communicate with the student. 

14. Plans and prepares for each lesson. 

The participants were also asked to group their checked items into 
areas of a similar nature or class and suggest a group title. This was 
acc ~>mpl i shed by first stating a title and then listing the numbers of their 
cherked elements which they felt belonged under this title. 

The results of the rccond round were studied by the researchers. 
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First, the elements checked by the respondents as being necessary 
in a definition of effective college teaching were tabulated and a 
frequency count made. The frequency table for the 195 elements showed 
a decided break after the first 50 elements. These 50 elements became 
the material for round three. 

Efforts to analyze the groupings proved too difficult. The analysis 
was especially difficult after having selected 50 elements and disregard* 
ing the remaining 145 elements. It was decided that further analysis 
was impossible and the study proceeded to the third round. 

After duly thanking the participants for their previous efforts 
they were asked to examine the list of 50 most frequently checked 
elements and arrange them into natural groupings. In addition they were 
asked to provide a name representative of each group they selected. 

The results of round three were studied in an effort to isolate 
major categories associated with effective college teaching. 

The groupings were first analyzed as to content. The respondents 
were in unexpected agreement as to their organization of the elements, 
however, there were a wide variety of names assigned to the groupings. 

By compating elements within a group it was possible to match up the 
various respondents groupings into a final listing of seven major 
categories . 

Each category was given a mul t i - el omen t heading or name made up 
of those suggested by the participants. 

These tabulations were further analyzed and a final listing was 
prepared containing seven major categories. Each category was given the 
two most irequent names and contained the five most frequent elements. 




- 3 - 



In round four the participants were asked to select the name they 
favored for each category and also select the three elements within the 
category which they felt best described the category. They were further 
instructed to combine any of the major categories which they felt should 
be so combined and briefly explain their rational. 

The results of round four indicated that there should be five major 
categories. The major categories were identified by using the most 
frequently checked name plus the three most frequently checked elements. 
The results of this fourth round are presented as the final results of 
this part of the study and are found in Table II. 

TABLE II 

Major Categories of Effective Teaching 

1. Personal Characteristics: 

Fciendly, Mature, Enthusiastic 

2. Professional Qualities: 

Attitudes, Knowledge, Preparation 

3. Interpersonal Relationships: 

Approachable, Fair, Honest 

4. Technical Skills of Teaching: 

Methods, Grading, Organization 

3. Communication Skills: 

Rapport, Relevancy, Open 

The second major portion of this research effort wa3 to determine 
the relative importance of the five elements of the definition. 

The results of the Delphi Study were used to develop twenty-five 
fictitious profiles of faculty members. Each profile was a rating scale 
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consisting of five nine-point Likert type items. The profiles had the 



following form: 



1. Personal Characteristics: 1 23456789 

Friendly, Mature, Enthusiastic 

2. Professional Qualities: 123456789 

Attitudes, Knowledge, Preparation 

3. Interpersonal Relationships: 1 23456789 

Approachable, Fair, Honest 



4. Technical Skills of Teaching: 123456789 

Methods, Grading, Organization 

5. Communication Skills: 123456789 

Rapport, Relevancy, Open 



The profiles used are presented in columns 1-5 of Table III . 

Therefore, each faculty member (judge) made a judgment of the 
importance of the various items on the profiles. For example, if a 
particular judge felt that item number five, Communication Skills, 
was most important in determining promotions, he would tend to place 
profiles in Set Number 1 which had ahigh rating on item five. When 
the separation of the profiles was complete, each judge had associated 
a rank from 1-5 with each profile. 

In order to have high ratings on the profile associated with high 
judgments, the weight of the judgments were reversed before the data 
was tabulated. That is, profiles placed in Set Number 1 were given 
a value of 5, Set Number 2 a value of 4, etc. 
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PROFILE 



JUDGES 



w 





Personal Characteristic 


Professional Qualities 
and Attitudes 


Interpersonal 

Relationships 


Technical Skills 
of Teaching 


i Communication 
Skills 


1 Judge 1 

i 


Judge 2 


i 

| Judge 3 


ID 


1 


2 


3 


4 


r 


6 


7 


8 


1 


5. 


7. 


6, 


5. 


3 . 


2. 


3. 


3. 


2 


5. 


7. 


5. 


4. 


6 . 


5. 


1. 


3 


3 


5. 


6. 


7. 


5. 


4. 


5. 


i. 


3. 


4 


5. 


6 . 


5. 


4. 


7. 


4. 


2. 


5. 


5 


5. 


6. 


4. 


5. 


8. 


3. 


1. 


4. 


6 


5. 


4. 


7. 


6. 


5. 


2. 


4. 


1. 


7 


5. 


4. 


5. 


6. 


7. 


3. 


5. 


2. 


8 


8. 


8 . 


5. 


5. 


4. 


5. 


2 . 


5. 


9 


7. 


5. 


6. 


4. 


5. 


4. 


2. 


5. 


10 


7 . 


5. 


5. 


3. 


6. 


1. 


5. 


5. 


11 


7 . 


4. 


5. 


6. 


5. 


2. 


5. 


5. 


12 


7. 


5. 


4. 


5. 


6. 


2. 


2. 


4. 


13 


6. 


8. 


5. 


5. 


4. 


5. 


1. 


4. 


14 


6. 


5. 


5. 


7. 


4. 


4. 


4. 


2. 


15 


6. 


5. 


4. 


7. 


5. 


3. 


4. 


2. 


16 


6. 


3. 


5. 


5. 


7 . 


1 . 


5. 


5. 


17 


6. 


4. 


8. 


5. 


5. 


3. 


3. 


5. 


18 


3. 


7. 


6. 


5. 


5. 


2. 


3. 


4. 


19 


4. 


6 


5. 


7. 


5. 


5. 


1. 


1 . 


20 


4. 


5. 


6. 


7. 


5. 


4. 


3. 


2. 


21 


4. 


5. 


7. 


6. 


5. 


4. 


3. 


2. 


22 




5. 


5. 


8. 


6. 


5. 


4. 


1. 


23 


». 


5. 


6. 


7. 


4. 


4. 


2. 


1. 


24 


5. 


5. 


7. 


6. 


4. 


4. 


2. 


2. 


25 


5. 


5. 


3. 


7. 


6. 


1 . 


5. 


1. 



faculty members were sent a set of the twenty-five profiles with the 
following Instructions: 
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You are asked to separate the enclosed 25 fictitious profiles into 
five sets. Those individuals you feel most deserving of promotion 
are to be placed in set Number 1, the next most deserving are to 
be placed in set Number 2, and so on, until the last are placed 
in set Number 5. Please place at least one profile in each of 
the five sets. 

It is not necessary to rank them within each set; merely list 
the profile member(s) in each set. 

Multiple Linear Regression, see bibliography, was used to fine 
which of the Items on the profileswere being used to make the judgment 
and the order of priority placed on the items by a particular judge. 

In the regression model the judgments of a judge were used as the 
criterion and the five vectors of profile secies were used as predictors. 
Columns 6, 7, and 8 in Table III are three samples of judgments 
received. The predictors were dropped from the model one at a time 
and the contribution of each was tested with the F statistic. The 
contribution of a predictor was considered important if the Fratio 
was significant at the .05 level. The items that contributed sig- 
nificantly for a particular judge were ranked In the order of the 
associated F ratio, highest value being assigned a 1. Non-slgnlficant 
items were not ranked. These rankings are summarized in Table IV. 

Sixty-one judges particupated in the study. In most cases only one 
or two of the Items on the profile made a significant contribution to 
the prediction of a particular set of judgments. This Is apparent 
from the relative lack of 3*s, 4*s and 5's In Table IV. The average 
number of significant Items per judge was 2.38. 

Inspection of Table IV shows that about 38K of the judges considered 
item 2, Professional Qualities, to be the most important in deciding 
on faculty promotion. By combining rankings of 1 and 2 it can be seen 
that items 1, 3, and 5 were about equally applied by the judges. 
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These items are Personal Characteristics, Interpersonal Relationships, 
and Communication Skills. Item 4, Technical Skills, was considered 
to be of least impor-anco to this group of faculty members in deciding 
promotions . 



TABLE IV 

Rankings of Predictor Variables 



Item 


1 


2 


Rank 

3 


4 


5 


ms 


Total 


Personal Characteristics 


13 


7 


2 


5 


1 


33 


110 


Professional Qualities 


23 


4 


7 


2 


0 


25 


156 


Interpersonal 

Relationships 


5 


12 


4 


3 


4 


33 


95 


Technical Skills 


7 


4 


5 


5 


2 


38 


78 


Communication Skills 


9 


11 


6 


1 


3 


31 


112 



In brief summary, the modified Delphi portion of the study 
selected five major characteristics of effective teaching. These five 
major characteristics are: 

Personal Chat ac teristics 
Professional Qjalities 
I n terprersonal Relationships 
Technical Skills of Teaching 
Communication Skills 

These charac teristics were used to develop artificial profile's 




for use in the last part of the study. 
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The profiles were sorted by 61 judges as tp preference for 
promotion . 

These sorts were analyzed by multiple linear regression and it 
was found that category 2, Professional Qualities, was deemed most 
important. Categories 1, Personal Characteristics; 3, Interpersonal 
Relationships; and 5, Communica t ion Skills were approximate ly euqlly 
valued as second in importance. Characteristic 4, Technical Skills 
was least important. 

The researchers feel that this study has demonstrated the usability 
of the Delphi technique in areas that heretofore have been difficult 
to study due to problems of definition. The study also points up 
that the evaluation of the effectiveness of teaching must take into 
consideration variable weighting of the predictor variables. 
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